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ABSTRACT 


The  purpose  of  Contract  NOnr  444l-(00)  -is  to  provide  an  empirical  test  of 
an  information  indexing  approach  to  the  assessment  of  user  requirement ,  i.e,, 
the  information  requirements  placed  upon  intelligence  and  surveillance  systems 
acting  in  support  of  command-control  systems „  This  report,  which  draws  heavily 
from  work  completed  within  a  classified  information  (secret)  context  under  Contract 
NOnr  2818- (00) s  presents  the  information  indexing  approach  and  contrasts  it 
with  two  existing  alternatives „  Also  included  are  illustrative  examples  of  means 
by  which  each  of  the  three  approaches  can  be  implemented  in  practice „ 


The  method  of  testing  the  validity  of  critical  assumptions  underlying 
the  information  indexing  approach  will  be  discussed  in  a  second  report  of  the 
completed  under  the  same  contract ,  Briefly s  the  approach  followed  involves 
use  of  an  experimental  task  which  is  a  simplified  analog  of  a  number  of  military 
situations 3  eaga ,  an  antisubmarine  warfare  (ASW)  situation  and  an  antiaircraft 
warfare  situation  (AAW)0  The  context  of  the  problem  is  such  that  it  allows  both 
objective  and  subjective  determinations  to  be  made  of  the  worth  of  various  types 
of  information0 
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I.  DISCUSSION 


A.  STATEMENT  OF  PROCEDURE 

The  problem  approached  here  is  that  of  deriving  "information  require¬ 
ments"  of  operational  surveillance  networks  when  such  networks  act  in  support 
of  executive  decision-making  or  command- control  operations.  In  most  cases 
these  "requirements"  can  be  interpreted  as  a  set  of  constraints  placed  upon 
the  supporting  system  by  users  or  consumers,  who  are  normally  men;  thus, 
both  human  factors  and  systems  analysis  problems  are  involved. 

Such  assessment  problems  have,  in  the  past,  been  attacked  by  one  or 
two  possible  approaches.  The  first  of  these  may  be  termed  a  rational  procedure 
by  which  available  quantitative  techniques  such  as  information  theory  and 
dicision  theory  are  applied  directly  to  the  problem.  Thus,  a  requirement 
may  be  phased  in  terms  of  a  stated  number  of  bits  of  information  or  an 
expected  value  of  x  units.  The  second  or  unique  solution  approach  attempts 
to  identify  those  items  of  information  whose  presence  is  both  necessary  and 
sufficient  for  the  implementation  of  an  assigned  mission  or  function.  Usually 
requirements  obtained  by  use  of  the  unique  solution  procedure  are  specified 
in  the  form  of  lists,  whose  items  may  or  may  not  be  arrayed  with  respect  to 
some  variable,  e,  g„  ,  in  order  of  their  time  priority  or  need.  A  third  alter¬ 
native,  termed  an  information  index  approach,  is  presented  with  the  aim  of 
bridging  the  gap  between  the  alternatives  listed  above.  The  indexing  approach 
has  as  its  basic  precept  the  application  of  weightings  which  are  derived  from 
"importance"  or  utility  values  and  the  quantity  associated  with  certain  outputs 
of  an  information  system.  These  outputs  are,  in  the  aggregate,  indicative 
of  system  performance. 

This  is  not  an  attempt  to  provide  a  completely  exhaustive  survey  of 
approaches  and  methods  of  implementing  these  approaches.  It  simply  attempts 
to  point  out  differences  in  three  approaches  to  the  same  general  problem; 
approaches  which,  while  they  are  related,  are  sufficiently  disparate  to  yield 
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widely  different  formats  for  re  suits —  for  mats  which  dictate  important 
restrictions  on  the  range  of  application  of  the  results. 

Before  reviewing  the  major  characteristics  of  each  technique  and  the 
type  of  problems  to  which  they  are  applicable,  it  is  necessary  to  clarify 
the  meaning  of  the  term  "requirement."  Basically,  when  discussing 
"requirements,"  correlations  are  mentioned*  i.  e.  ,  the  relationship  between 
(a)  certain  types  and  amounts  of  information,  and  (b)  decision  adequacy. 
Therefore,  the  following  definition  can  be  offered:  An  information  require¬ 
ment  is  that  amount  and/or  those  types  of  information  produced  by  the 
supporting  operational  network  which  allows  the  decision-maker  to  attain 
a  proficiency  level  €. 

The  above  definition  implicity  assumes  that  a  requirement  imposed 
upon  an  operational  surveillance  network  is  directly  related  to  an  assess¬ 
ment  of  that  network’s  output.  Stated  another  way,  it  is  possible  to  discuss 
information  in  terms  of  the  difference  it  makes  in  command  performance 
when  it  is  present  or  absent.  If  this  difference  is  measureable  in  some  way, 
a  value  tag  can  be  attached  to  that  item  of  information  and  a  requirement 
can  be  stated  in  terms  of  a  set  of  items  having  a  stated  value.  In  short, 
what  is  "required"  or  "desired"  is  some  level  of  executive  performance 
including  the  antecedent  conditions  which  permit  its  being  reached. 

In  the  discussions  provided  below,  assessment  is  intimately  involved 
in  requirement  specification.  Therefore,  there  is  considerable  interest 
in  how  this  assessment  is  made  and  how  it  serves  to  identify  what  the  user 
needs  in  order  to  perform  his  job  adequately. 

B.  THE  MAJOR  ALTERNATIVES 

The  first,  or  rational,  approach  can  be  applied  to  systems  problems 
when  considerable  a  priori  knowledge  is  available  concerning  both  the 
operational  network  and  the  executive  decision-making  system.  For 
example,  if  information  theory  is  to  be  used,  it  is  necessary  to  catalogue 
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the  universe  of  information  passing  through  the  network  into  dimensions 
and  to  assign  values  on  these  dimensions  which  are  within  the  message 
space.  Then,  the  bit  content  of  the  information  processing  system  can 
be  measured  under  a  wide  range  of  conditions.  However,  even  this  step 
is  only  part  of  the  problem  because  interest  is  primarly  centered  about  the 
universe  of  decision  alternatives  and  the  way  in  which  information  supplied 
by  the  network  serves  to  reduce  uncertainty  concerning  the  appropraite 
course  of  action  to  take.  Thus,  the  executives'  (or  commanders')  decision 
space  must  be  similarly  treated.  The  resulting  correlation  would  then 
occur  between  bits  of  informa.tion  in  the  message  space  and  bits  of  information 
in  the  decision  space*. 

Decision  theory  can  also  be  used  to  implement  the  rational  approach. 
Here,  investigators  focus  on  the  command  or  executive -output  and  measure 
its  worth  or  value.  Then,  if  the  "decision  rules"  are  known  (that  is  if  it  is 
known  what  decisions  are  made  in  the  light  of  certain  information)  the 
information  input  is  manipulated  over  a  wide  range  of  conditions  and  a  choice 
is  made  of  that  manipulation(or  those  manipulations)  which  reached  the 
desired  proficiency  level  or  output  level.  Another  related  use  of  decision 
theory  has  been  suggested  by  Schreiber  (1961)  whereby  one  selects  those 
items  of  information  which  produce  the  largest  variations  in  output  utility. 


*The  terms  "message  and  decision  spaces"  as  used  here  refer  to  the 
universe  of  possible  information  and  the  universe  of  possible  decisions. 
Special  concern  is  centered  about  the  extent  to  which  elements  within  the 
information  and  decision  spa.ces  can  be  described  in  analytical  terms,  viz.  , 
as  coordinates  in  a  multidimensional  space.  However,  in  order  for  this 
to  occur,  the  basic  variables  involved  and  their  interrelationships  need  to 
have  been  identified.  Otherwise,  the  space  cannot  be  dimensionalized.  In 
addition,  all  elements  need  to  have  the  property  of  being  able  to  be  measured 
in  these  dimensions.  If  these  conditions  are  not  met,  the  spaces  can  only 
be  categorized  into  mutually  exclusive  sets  and  subsets  according  to  some 
common  underlying  criterion  such  as  message  content. 
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The  second  technique  mentioned,  the  unique  solution  approach,  does  not 
possess  the  metrical  refinement  gained  through  the  use  of  decision  theory  or 
information  theory.  It  is  used  primarily  in  situations  where  a  priori  knowledge 
is  at  a  bare  minimum  --  such  as  the  case  in  planning  a  new  system.  By  means 
of  survey  or  review  methods,  investigations  follow  what  is  basically  a  job  analysis 
or  job  specification  approach  and  attempt  to  identify  information  which  "should 
be"  transmitted  by  the  network.  Thus,  an  implicit  assumption  is  usually  made 
that  the  proficiency  level  desired  has  a  fixed  value  --  or  at  least  a  fixed  minimum 
value.  The  listed  items  can  thereby  serve  as  a  stipulation  of  the  conditions  necessary 
and  sufficient  for  reaching  this  value.  Such  an  approach,  which  is  well  illustrated 
by  the  work  of  Vaughan,  et  al.  ,  (i960),  has  the  disadvantage  of  not  being  readily 
applicable  when  the  proficiency  level  desired  is  likely  to  vary  or  when  the  task  is 
one  of  assessing  alternative  networks  instead  of  stipulating  what  the  networks 
should  be  handling.  However,  it  is  applicable  to  cases  where  the  information 
units  are  not  additive,  that  is  where  units  of  one  type  of  information  do  not  have 
some  equivalence  relation  to  units  of  another  type  of  information  --a  condition 
which  precludes  use  of  the  rational  approach. 

The  third,  or  information  index  approach,  will  be  discussed  most  fully  in 
this  report;  .  It  is  applicable  to  situations  where  the  general  nature  of  an 
operational  network  is  known  but  where  certain  crucial  aspects  of  the  decision 
process  remain  unspecified.  In  such  cases  the  different  types  of  information 
available  can  only  be  categorized  (sorted  by  content);  dimensionalizing  the 
information  space  is  impossible.  Also,  decision  rules  are  not  at  all  fixed  and 
are  usually  a  matter  of  individual  judgment  (executive  and  command  experience)., 
Finally,  the  decision  space  is  not  dimensionalized  either,  because  the  executive 
may  at  a  later  date  have  new  courses  of  action  available  to  him. 

In  such  situations,  which  are  illustrated  best  by  military  intelligence  net¬ 
works,  there  is  a  need  to  evolve  a  quantitative  assessment  technique  which  can 
be  used  to  evaluate  the  products  of  a  series  of  proposed  networks  available  now 
or  in  the  future.  Usually,  such  a  goal  can  be  accomplished  by  the  use  of  a  two 
stage  process.  First,  users  are  asked  to  derive  estimates  of  value  for  various 
types  of  information  under  a  wide  range  of  operating  conditions.  Then,  these 
hypothetical  utility  values  are  checked  against  objective  outputs  from  the  executive 
system  to  insure  that  the  utility  values  are  valid  in  actual  practice.  An  alternative 
to  this  sequence  is  one  whereby  the  value  index,  which  is  in  reality  an  intermediate 


criterion,  is  computed  in  a  series  of  specific  command  situations  and  generalized 
later.  In  both  cases,  however,  the  approach  is.  analogous  to  the  one  employed  in 
multiple  regression;  an  index  value  is  expressed  as  a  function  of  a  series  of 
variables  each  of  which  has  some  importance  weighting.  What  makes  the  approach 
more  complicated  than  traditional  regression  techniques  is  the  fact  that  decision 
rules  are  so  poorly  specified.  Therefore,  some  means  of  circumventing  this 
obstacle  has  to  be  devised  before  characteristics  of  the  information  supplied  can 
be  related  unambiguously  to  characteristics  of  the  resulting  decisions. 


C.  DISCUSSION  OF  THE  ALTERNATIVES 

As  in  the  case  with  most  classification  schemes,  separation  of  the  three 
approaches  outlined  above  is  somewhat  arbitrary.  In  reality,  all  of  the  three 
are  interrelated  to  some  extent.  For  example,  the  rational  and  information  indexing 
approaches  presuppose  the  existence  of  knowledge  such  as  is  obtained  in  the 
unique  solution  method,  i.  e.  ,  that  the  operational  system  available  has  been 
specified  to  a  considerable  extent.  Furthermore,  decision  theory  in  some  cases 
can  be  used  to  supplement  the  findings  of  the  unique  solution  approach  for  the 
purpose  of  generating  performance  implications  for  these  "necessary  and  sufficient" 
conditions  even  when  the  information  space  is  not  dimensionalized.  Also,  the  use 
of  a  decision  theory  measure,  utility,  is  incorporated  into  the  information  indexing 
approach  in  an  attempt  to  allow  quantification  when  decision  rules  do  not  exist. 

If  they  do  exist,  and  if  the  information  space  can  be  catalogued  only  by  message 
content,  some  combination  of  the  first  and  third  approaches  evolves.  In  short, 
in  addition  to  the  use  of  each  approach  alone,  combinations  of  approaches  are 
possible. 

Granting  that  there  are  complications  arising  in  practice  which  can  make  it 
difficult  to  distinguish  between  the  three  basic  techniques,  enough  important 
differences  remain  to  make  the  above  classification  useful  in  systems  analysis 
problems.  These  differences  generally  arise  from  two  sources:  (a)  the  nature 
of  the  problem  under  investigation  and  (b)  the  use  to  which  the  analysis  will  be 
put.  These  differences  can  be  illustrated  by  presuming  that  an  investigator  begins 
his  work  by  asking  a  series  of  questions,  the  answers  to  which  determine  what 
technique  should  be  or  can  be  used.  These  are  listed  below  along  with  a  discussion 
of  what  occurs  when  the  answer  is  affirmative  or  negative. 
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1.  Has  the  information  passed  through  the  operational  network  been 
specified  and  categorized? 

a.  If  not,  the  primary  problem  is  to  accomplish  this  task  by  means 
of  something  similar  to  the  unique  solution  approach.  This  technique  presumes 
that  is  is  possible  to  stipulate  some  desired  level  of  command  performance  which 
can  be  used  as  a  reference  point  when  obtaining  what  the  network  should  or  would 
handle.  *  Usually,  this  approach  terminates  with  the  production  of  lists  of  things 
to  be  processed  by  the  network. 

These  results  can  be  used  to  evaluate  the  output  of  proposed  systems 
(hence  the  system  itself)  in  a  checklist  fashion.  A  further  refinement  can  be 
added  by  weighting  each  item  on  the  lists  according  to  some  criterion  of  impor¬ 
tance  (cf.  ,  Vaughan,  et  al.  ,  I960).  When  this  latter  step  is  taken  the  unique 

solution  approach  begins  to  approximate  results  gained  by  use  of  the  information 
indexing  approach;  however,  there  still  is  a  difference  remaining  because  if 
the  desired  command  performance  level  is  changed  (the  reference  point  mentioned 
above),  there  is  no  guarantee  that  the  weightings  will  remain  constant.  Therefore, 
a  number  of  iterations  of  the  unique  solution  approach  may  be  necessary. 

Decision  theory  can  also  be  used  in  conjunction  with  the  unique 
solution  approach  to  determine  exactly  what  level  of  performance  stems  from 
the  use  of  the  proposed  solution  and  what  happens  when  the  solution  is  only 
approximated  or  not  used  at  all.  Such  a  possibility  is  greatly  enhanced  if  one 
has  at  hand  the  decision  rules  relating  information  input  to  resulting  command 
choices.  If  such  rules  are  not  available,  large  scale  simulations  and/or  system 
exercises  are  needed. 


’^While  it  is  theoretically  possible  to  derive  a  solution  for  a  series  of  reference 
points  at  one  time,  practical  limitations  normally  preclude  such  a  possibility. 
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Thus,  the  unique  solution  approach  is  appropriate  when  the 
characteristics  of  the  operational  network  are  not  well  known.  Its  primary 


function  is 

to  identify 

the  nature  of  the  information  that  should  or  would  be  pro- 

duced  by  the  network. 

In  addition,  further  refinements  can  be  added  to  indicate 

what  satisfaction  of  these  requirements  means  in  terms  of  command  system  per 

formance. 

However , 

practical  considerations  normally  preclude  following  this 

approach  when  a  series  of  executive  performance  levels  are  to  be  studied. 


b.  If  the  condition  specified  above  is  satisfied,  a  second  question  is 
proposed. 

2.  To  what  extent  can  the  information  space  and  the  decision  space  be 
measured  and  related? 

a.  If  the  decision  and  information  spaces  can  only  be  sorted  into 
mutually  exclusive  categories,  while  the  decision  rules  relating  the  two  remain 
vague,  the  information  indexing  approach  is  applicable.  This  technique,  which  is 
discussed  at  length  below,  assumes  in  its  most  general  form  that  some  type  of 
equivalence  relation  exists  between  the  various  types  of  information.  If  this  is 
not  sufficiently  compatible  with  reality,  the  unique  solution  technique  outlined 
above  can  be  followed  for  each  type  of  information  by  itself.  However,  at  best, 
the  information  indexing  approach  only  approximates  what  would  be  obtained  by 
use  of  the  rational  approach. 

b.  If  the  decision  and  information  space  can  be  dimensionalized  and  all 
items  of  information  passing  through  the  system  have  some  value  on  each  dimension, 
rational  techniques  can  be  used  even  when  decision  rules  are  not  completely  speci¬ 
fied.  However,  the  extent  of  manipulation  required  normally  is  too  great  for 
experiments  involving  command  and  executive  personnel.  Therefore,  while  the 
approach  is  applicable  in  a  theoretical  sense,  practical  considerations  often 
eliminate  it  from  consideration.  When  the  decision  rules  are  known,  however, 

Monte  Carlo  techniques  can  be  used  to  establish  the  necessary  correspondence. 

D.  THE  INFORMATION  INDEXING  APPROACH 

Without  attempting  to  escape  blame  for  inadequacies  of  methodological  rigor 
contained  in  succeeding  portions  of  the  paper,  the  authors  wish  to  underscore  the 
fact  that  this  technique  is  not  a  device  of  their  personal  invention.  Caldwell,  et  al.  , 
(1959)  employed  an  analogous  approach  to  the  problem  of  evaluating  the  output 
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of  an  Army  intelligence  system.  However,  the  authors  were  not  concerned 
with  examining  their  approach  as  much  as  the  results  gained  from  the  use  of 
it;  therefore,  the  present  discussion  overlaps  only  slightly.  In  addition,  the 
technique  draws  heavily  on  utility  measurement  theory  prepared  by  Davidson, 

Siegel,  and  Suppes  (1955). 

The  key  to  evolving  a  method  which  promises  to  be  more  than  an  interesting 
mental  exercise  lies  in  the  initial  "structuring"  of  the  problem  --  primarily  as 
it  affects  the  identification  of  "units  of  information.  "  Often  operating  systems 
have  "natural"  units  present  --  standardized  messages  for  example  --  which 
bear  some  unspecified  relationship  to  an  investigator's  basic  units  of  information. 
When  the  bit  measure  cannot  be  employed  because  of  practical  limitations,  it 
is  necessary  to  offer  some  substitute  since  "natural"  units  vary  widely  in  the 
amounts  of  information  contained.  In  such  cases,  it  is  often  possible  to  use  an 
approximation  technique  whereby  one  separates  natural  units  into  items  of 
information. 

The  definition  of  an  item  must  be  adjusted  to  specific  system  requirements; 
therefore,  it  is  difficult  to  define  in  general.  In  the  case  of  surveillance  systems, 
however,  an  item  can  be  defined  as  an.  observation  made  over  a  specific 
uin  'Corrupted  '.m  vai  of  Vn  .me  -t  a  particular  physical  thing  or  an  observable 
attribute  or  characteristic  of  that  thing,  the  observation  being  made  by  a  single 

T— - -  - - - - - — - — 

sensing  device  (including  man).  The  criteria  for  deciding  if  one  or  two  items 

- - ~~ -  -  - .  r 

are  present  therefore  become  as  follows:  (l)  is  the  same  sensor  being  used 
as  a  source  of  information?  (2)  is  the  object  or  characteristic  being  observed 
the  same?  and  (3)  is  the  time  of  observation  the  same?  For  convenience  (which 
will  become  evident  later)  one  additional  criterion  is  suggested:  (4)  are  the  format 
and  content  of  the  information  in  the  item  constant? 

At  this  point  in  the  approach,  the  investigator  has  subdivided  the  information 
produced  by  an  operational  surveillance  network  --  or  a  portion  of  it  --  oyer  a 
given  time  interval  into  a  set  of  messages  correlated  with  the  existing  natural 
units.  These  messages  can  then  be  further  subdivided  into  items  in  most  cases; 
or,  if  he  is  indeed  fortunate,  the  investigator  can  use  the  familiar  bit  measure  and 
ignore  the  remainder  of  this  approach  which  is  based  upon  the  assumption  that 
the  rational  treatment  is  inapplicable. 


If  the  above  sequence  can  be  accomplished  in  sufficiently  precise 
form,',  it' becomes  comparatively  simple  to  complete  the  approach.  First, 
the  items  of  information  produced  by  the  networks  are  examined  vyith  respect 
to  their  content,  and  classified  into  a  set  of  mutually  exclusive  exhaustive 
categories.  Each  item  is  assigned  to  one  and  only  one  category.  Now,  if 
the  utility  of  these  content  categories  can  then  be  measured  by  means  of  the 
methods  of  implementation  to  be  explained  later,  the  basic  outline  of  the 
approach  becomes  evident. 


For  example,  assume  that  a  highly  simplified  case  exists;  one  . 
where  the  network  contributes  items  of  information  which  are  highly  valid. 
It  is  now  possible  to  compute  an  index  of  worth  of  the  network  output  being 
evaluated  by  the  use  of  Equation  1. 


N 

n 

B  =  Z 

S 

3=1 

i=l 

Wi  °j 


(1) 


where: 

B.  =  the  index  of  worth  attributed  to  the  output  of  a  supporting 
operational  system  estimated  on  the  basis  of  considering 
user  requirements, 

N  =  the  number  of  mutually  exclusive  content  areas  into  which 
items  of  information  fall, 

n  =  the  number  of  items  contained  in  a  message, 

W  =  the  number  of  items  belonging  to  the  j^1  content  class 
contained  in  the  i^1  message, 

a-  =  the  average  value  to  the  user  of  an  item  of  information 
3 

in  the  jth  class. 

Thus,  the  above  index  is  nothing  more  than  a  consideration  of  both  the  amount 
of  information  produced  by  a  network  and  the  worth,  or  value,  of  this  information. 
It  is  an  attempt  to  place  a  price  tag  on  items  of  information  produced  by  an 
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operational  network  during  a  given  .time  interval.  The  equation  must  be  modi¬ 
fied  when  the  network  produces  information  which  is  not  always  highly  accurate. 

This  can  be  accomplished  by:  (1)  adding  a  validity  factor  jzh,  which  can  be 

1  til 

defined  as  the  estimated  probability  that  the  information  in  the  i  item  is 
correct;  and,  (Z)  by  adding  a  second  summation  which  subtracts  from  the 
value  of  the  correct  information  the  loss  in  value  accruing  as  a  result  of  the 
incorrect  information. 


If  the  risk  situation  in  which  the  decision-maker  operates  is  symmetrical, 
i.  e.  ,  if  the  magnitude  of  a  gain  associated  with  a  particular  information  item 
equals  the  magnitude  (disregarding  sign)  of  the  loss  associated  with  that  item 
being  incorrect,  the  equation  can  be  represented  as  follows: 

N  n  N  n 

I  =  S  2  W.a.  d.  -  2  2  W.a.  (1  -  d.)  (2) 

j  =  i  i=l  1  3  3  j=l  i=l  1  3  3 


where 


I  =  the  index  of  worth  attributed  to  the  output  of  a  supporting 

operational  system  estimated  on  the  basis  of  considering  user 
requirements  including  validity. 

<J>.  =  the  validity  attached  to  the  item. 

j 

W.,  a.,  n,  and  N  are  defined  as  above, 
i  J 

E.  THE  MEANING  OF  VALUE 

It  is  apparent  that  the  utility  term  in  equation  1  and  2  is  of  particular 
significance.  Ideally,  this  term  would  be  determined  by  an  objective  cost 
analysis.  For  example,  consider  a  military  situation  which  is  in  doubt 
and  may  have  one  of  two  values,  A  or  not  A(A),  with  a  priori  probabilities 
of  p(A),  p(A).  Since  these  states  are  mutually  exclusive  and  collectively 
exhaustive,  the  following  relationship  exists 

P(A)  +  1(A)  =1  (3) 
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Consideration  must  be  given  to  errors  of  the  two  possible  types,  defined 
as  follows: 

Type  1  error  =  the  presumption  that  the  state  is  A  when  it  is 
actually  A . 

Type  2  error  =  the  presumption  that  the  state  is  A  when  it  is 
actually  A . 

Assumptions:  Assume  that  the  costs*  of  the  errors  of  the  two 
types,  Cx  and  C2,  are  known,  or  can  be  estimated,  in  terms  of  loss  of 
military  objectives,  men,  and  equipment.  This  assumption  is  implicit  in  any 
decision  rule  for  military  activities.  That  the  prior  probabilities,  p(A)  and 
P(A),  are  known  need  not  be  stated  as  an  assumption  since  one  can  act  upon 
the  premise  that  these  constitute  prior  "degrees  of  belief"  on  the  part  of  the 
military  commander. 

Decision  Rules:  There  are  two  basic  types  of  situations  to  be  considered: 

1.  The  a  priori,  no -information  case  where 

Ex,  or  the  Expected  cost  of  assuming  "A"  = 

CxP(A),  and  E2,  or  the  Expected  cost  of  assuming 
"A"  =  C2P(A). 

Assume,  in  the  following  development,  that  the  rational  commander  always 
follows  the  "expected  value"  strategy  so  as  to  minimize  "expected  cost.  " 
Hence,  if  CxP(A)  >  C2P(A),  then  Ej  >  E2,  so  he  will  act  upon  the  premise  that 
A  is  the  true  state  of  affairs.  If  E2  >  E1;  he  will  act  upon  the  premise  that 
A  is  the  true  state  of  affairs. 


*"Cost  of  an  error"  refers  to  net  cost  to  the  command  system;  e.  g.  ,  if  the 
true  state  of  affairs  is  A  and  the  commander  acts  upon  the  premise  that  it 
is  A,  the  cost  of  the  error  is  the  cost  of  the  outcome  under  the  wrong  decision 
minus  what  the  cost  of  the  outcome  would  have  been  if  he  had  made  the  right 
decision. 


2.  The  a  posteriori,  partial  information  case: 


Now  assume  that  an  item  of  information  is  provided  to  the  commander , 
and  that  this  item  has  two  values,  B  and  not  B(B),  during  a  given  period  of 
observation.  For  example  B  could  represent  the  occurrence  of  a  "blip"  on  a 
sensor  display,  and  B  its  nonoccurrence,  while  state  A  could  represent  the 
presence  of  an  enemy  submarine,  and  A  its  absence. 


Further,  suppose  that  the  event  of  B  has  a  known  probability  of  occurrence, 
p  (B/A),  in  the  absence  of  the  submarine,  (due  to  the  presence  of  sea  life, 
oceanographic  irregularities,  etc,),  and  a  greater  probability,  p(B/A),  of  occurrence 
in  the  presence  of  the  submarine.  Classical  signal-detection  theory  adapts 
Bayes1  Theorem,  which  states: 

p(A)  p(B/A) 

p(A/B)  =  -  (4) 

P(A)p(B/A)  +  P(A)p(B/A) 


This  is  the  expression  for  the  a  posteriori  or  contingent  probability 
of  A,  the  presence  of  the  submarine,  which  would  represent  the  rational  com¬ 
mander's  degree  of  belief  in  the  submarine's  presence  after  having  seen  the 
sensor  "blip,  "  B. 

Analogously,  we  derive 


p(A)  p(B/A) 

p(a/b)  =  "t  ’  — — - — — — - r —  (5) 

P(A)  p(B/A)  +  P(A)  p(B/A) 


This  represents  the  rational  commander's  a  posteriori  belief  in 
the  submarine's  presence  given  that  the  sensor  fails  to  yield  a  "positive  " 
return,  "B";  i,  e,  ,  "B"  is  the  actual  state  of  affairs. 


In  the  case  of  no  information,  the  expected  cost  (Ec)  to  the  system 
was  simply 


Cx  p(A- 


Ec  =  min. 


C  2  p(A) 


(6) 


! 


The  expected  cost  to  the  system  in  the  presence  of  the  new  item 
of  information  is 
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Ec'  =  p(B)min  Cj  p(A/B) 

C2  p(A/B) 


+  1  -  p(B)  min.  Cj  p(A/B) 

|  C2  p(A/B) 


Thus,  the  difference  value,  Ec  -  E<4,  represents  the  average  or 
weighted  worth  of  the  information  about  the  state  of  "B--not  B"  to  the  decision¬ 
maker.  It  corresponds  to  the  a.  term  in  Equation  (2). 

J 

The  foregoing  arguments  can  be  readily  generalized  to  cases  where 
the  military  state  of  affairs,  and/or  the  information  given,  may  each  have  more 
than  two  possible  values.  While  the  illustrative  example  was  drawn  from,  original 
detection  theory,  the  same  principles  would  obviously  apply  to  any  item  of  relevant 
tactical  or  strategic  information.  However,  it  is  apparent  that  a  formulation  of 
utility  (a-j)  in  these  terms  requires  the  presence  of  considerable  foreknowledge, 
a  condition  rarely  satisfied  in  a  wide  range  of  intelligence  situations.  For  this 
reason,  less  complete  treatment  needs  to  be  adopted  in  certain  cases  using 
the  modes  of  implementation  described  next. 


F.  METHODS  OF  IMPLEMENTATION 

The  preceding  discussion  considered  three  possible  approaches  which  can 
be  used  to  derive  user  requirements.  Attention  will  now  be  focused  upon  an 
enumeration  and  description  of  the  methods  which  are  available  to  implement 
each  approach.  Each  method  of  implementation  is  aimed  at  relating  decisions 
to  the  information  inputs  of  the  command  system.  In  the  case  of  the  unique 
solution  approach,  the  decision  is  held  constant  while  information  input  is 
varied  in  order  to  determine  what  is  "needed"  for  the  decision  to  be  made.  For 
the  other  two  approaches,  the  information  input  is  manipulated  and  observations 
are  made  regarding  the  command  systems  output  in  order  to  relate  the  two 
factors  over  a  wide  range  of  operating  conditions.  In  either  case,  the  "mani¬ 
pulations"  may  be  made  by: 
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1. 


the  thought  processes  of  experienced  command  personnel; 

2.  gaming  or  other  simulations; 

3.  reconstruction  of  military  engagements  to  determine  what 
information  was  needed  and  how  it  was  used; 

4.  experimental  changes. 

In  determining  information  requirements  in  the  field  of  intelligence,  the 
following  modes  or  methods  of  implementation  seem  applicable; 

Method  1.  Intuitive  Judgment.  Experienced  military  personnel  who 
are  considered  experts  in  the  action  routines  of  the  command  system  are 
asked,  either  in  an  interview  situation  or  via  a  questionnaire,  to  deduce  the 
consequence  of  the  presence  or  lack  of  selected  intelligence  data  or  the  impor¬ 
tance  of  various  intelligence  items.  This  process  is  usually  repeated  for  several 
situations. 

Method  2.  Gaming  Techniques.  This  method  covers  a  wide  spectrum  of 
activities  ranging  from  the  hand-played  war  games  to  pure  simulation.  In  the 
hand-played  war  game,  decisions  are  made  by  humans  (as  players  or  as  umpires) 
and  the  outcomes  of  these  decisions  are  hand-recorded  and  analyzed  for  specific 
situations  of  interest.  In  a  machine  assisted  game  some  portions  of  the  routine 
activities  of  the  games  (the  recording  and  analysis)  are  handled  by  a  computer 
while  human  players  and  umpires  continue  to  participate  as  decision-makers. 

If  a  complete  set  of  rules  for  players  and  umpires  is  devised  the  game  can  be 
completely  machine  operated.  In  this  type  of  game  the  tactics  and  weapon 
parameters  of  both  sides  are  computer  performed,  and  the  game  is  played  by 
the  computer  with  a  readout  indicating  the  results  of  the  battle.  Data  derived 
from  these  techniques  are  usually  evaluated  against  a  criterion  of  threat  or 
survival. 

Method  3.  A  Posteriori  Investigations.  This  method  permits  after  the  fact 
evaluation  of  specific  information  inputs  to  the  command  system  as  a  function  of 
the  decisions  made  therefrom  in  cases  where  characteristics  of  both  the  informa¬ 
tion  and  the  decision  can  be  specified.  There  are  several  alternative  ways  in 
which  to  obtain  data  through  this  technique.  Specifically,  it  is  possible  to 
evaluate  the  recorded  messages  received  by  command  personnel  during  field 
exercises,  or  to  determine  the  relationship  of  actual  intelligence  to  successful 


decision-making  in  past  battles  through  historical  reconstructions.  Judgments 
of  value  in  this  case  would  use  the  final  outcome  of  the  decisions  as  the  criteria; 
the  final  outcome  being  determined  from  historical  documents. 

Method  4.  Formal  Experiments.  This  method  allows  for  the  development 
and  control  of  specific  parameters  related  to  the  actual  situation  of  interest  in 
order  to  state  within  a  certain  level  of  confidence  what  information  is  required 
by  a  decision-maker  in  that  situation. 

The  following  discussion  presents:  (l)  the  advantages  and  disadvantages 
associated  with  each  mode;  and,  (2)  illustrative  examples  from  available 
literature  on  how  these  methods  have  been,  or  might  be,  adapted  to  the  three 
alternative  approaches. 

G.  SELECTIVE  RESEARCH  REVIEW. 

The  intuitive  judgment  method  has  been  modified  to  serve  various  needs 
and  is  frequently  described  in  the  literature.  Even  though.it  permits  gathering 
large  amounts  of  data  with  a  small  expenditure  of  funds,  the  results  obtained 
therefrom  can  be  challenged  on  the  grounds  that  they  may  be  biased  by  the 
perceptions  of  the  respondents.  Equally  as  important  is  the  consideration 
that  there  really  is  no  guarantee  that  a  respondent's  insights  will  be  analogous  to 
"real  life.  "  Finally  there  is  the  problem  of  analyzing  requirements  in  situations 
which  may  be  so  complex  that  they  transcend  human  capability  to  extrapolate 
intuitively. 

Caldwell  et  al.  ,  (1959)  conducted  one  of  the  earliest  large  scale  studies 
using  the  intuitive  judgment  method  to  implement  an  indexing  of  intelligence 
information.  This  work  was  concerned  with  evaluative  analysis  of  Army  tactical 
intelligence.  The  ultimate  goal  of  the  study  was  to  obtain,  through  an  intuitive 
evaluation  of  intelligence  estimates  from  a  large  group  of  experienced  Army 
officers,  an  algebraic  model  which  would  make  possible  an  evaluation  of  the  "worth1 
of  various  items  of  intelligence  in  relation  to  the  value  of  an  intelligence  estimate. 
Caldwell  reports  that  four  questionnaires  were  used  to  elicit  the  responses  of 
the  officers  from  the  Army  Command  and  General  Staff  College.  These  question¬ 
naires  referred  to  three  specific  situation  reports  and  were  concerned  with: 

(l)  the  kind  of  information  an  officer  considered  important  in  judging  the 
intelligence  estimate;  (2)  the  kind  of  information  an  officer  considered  important 


in  judging  the  courses  of  action  presented;  and,  (3)  the  degree  of  confidence  with 
which  these  judgments  were  made.  Additional  analyses  were  performed  to  obtain 
indications  on  whether  there  was  a  more  simplified  procedure  for  evaluating 
intelligence  and  surveillance  systems.  The  model  developed  from  the  above 
study  make  it  possible  to  reach  some  interesting  conclusions  regarding  the 
expected  ''value'1  of  intelligence  on  a  particular  kind  of  terrain  in  an  offensive 
battle  where  conventional  weapons  were  used  by  a  specified  tactical  unit. 

An  example  of  this  method  being  used  to  implement  the  unique  solution 
approach  occurs  in  a  study  by  Bourne  et  al.  ,  (1961)  which  was  directed  toward 
determining  user  requirements  for  future  information  systems.  This  work 
was  done  within  the  context  of  information  storage  and  retrieval;  however,  the 
results  and  methodology  used  are  applicable  and  illustrative  for  our  purpose. 

A  survey  technique,  using  personal  interviews  was  employed  to  obtain  responses 
from  a  specific  user  population  in  order  to  determine  the  importance  of  various 
"user  requirements."*  These  user  requirements  were  reduced  to  seven  charac¬ 
teristics  which  were  assigned  weightings  by  a  method  of  rankings.  These 
characteristics  are  "generalizable"  to  a  variety  of  information  requirement 
problems  and  include:  (l)  minimum  time  to  get  relevant  information  to  the  user; 

(2)  minimum  amount  of  irrelevant  information  produced;  (3)  minimum  amount 
of  relevant  information  overlooked;  (4)  form  in  which  the  information  was 
received;  (5)  assurances  of  completeness  of  coverage;  (6)  ease  of  communication 
between  user  and  system;  (7)  indications  of  technical  competence  of  each  search 
product.  This  study  resulted  in  the  development  of  an  ordinal  list  of  importance 
for  -these  seven  characteristics  which  can  be  used  to  describe  the  optimum 
conditions  for  using  the  products  of  certain  information  systems. 

Gaming  techniques,  or  the  second  method  of  implementation,  possess  a 
somewhat  higher  degree  of  realism  and  objectivity  than  the  intuitive  judgment 
method  described  above.  There  are,  however,  disadvantages  associated  with 
each  of  the  techniques  covered  under  the  broad  category  of  Gaming  Methods.  For 
example,  when  hand  played  war  games  handle  problems  of  significant  detail, 
they  become  very  complex,  requiring  a  great  amount  of  input  data,  and  they 


*A  distinction  was  made  between  "users"  who  come  to  the  system  seeking 
information  and  "operators"  who  actually  work  and  maintain  the  system. 


usually  generate  a  great  amount  of  output  data.  This  type  of  game  is  usually 
slow,  requiring  a  great  deal  of  time  from  players  and  umpires  as  well  as  from 
those  who  prepare  the  rules  and  analyze  the  results.  The  machine-assisted 
game  solves  the  problem  of  manual  recording  and  analysis  of  the  output  data, 
but  it  does  require  a  great  deal  of  programming  and  "debugging'1  time  and  is 
still  slow  and  costly  on  a  per  "battle"  basis.  Pure  simulation  reduces  the 
time  per  game,  but  it  is  necessary  to  reduce  all  of  the  player  and  umpire 
decisions  to  a  comphrehensive,  all  inclusive  list  of  decision  rules. 

Research  using  this  mode  to  implement  a  combination  of  the  information 
indexing  and  rational  approaches  has  been  done  by  the  staff  of  the  Operations 
Research  Office  (ORO)  of  the  Johns  Hopkins  University.  Moss  and  Harris 
(1961)  or  ORO  were  the  first  to  simulate  an  Army  intelligence  system  in  order 
to  determine  what  the  optimum  mixture  of  pertinent,  complete,  timely,  and 
accurate  intelligence  information  really  is.  Since  a  large  number  of  variables 
enter  into  an  intelligence  information  simulation  system  and  since  it 
was  possible  to  quantify  (at  least  to  a  reasonable  degree)  these  variables,  it  was 
decided  that  a  computer  simulation  was  preferable  to  a  hand  played  or  computer- 
aided  simulation. 

Briefly,  the  method  employed  by  Moss  and  Harris  for  determining  this 
optimum  mixture  via  computer  simulation  was  as  follows: 

1.  An  area  of  terrain  was  broken  down  into  relatively  small  units 
in  which  various  features  such  as  altitude,  height  of  foliage, 
traversability,  etc.  were  noted  and  programmed  into  the  computer. 

This  minute  breakdown  was  also  made  for  the  other  secondary 
variables.' 

2.  The  locations  of  both  the  Red  unit  and  the  Blue  unit  were 
also  programmed  into  the  computer  and  certain  capabilities 
were  given  to  the  Blue  unit  for  acquiring  intelligence  informa¬ 
tion  about  the  "enemy"  (the  Red  unit). 

3.  The  information  about  the  Red  unit  was  varied  randomly  but 
with  control  in  terms  of  amount,  pertinence,  and  accuracy. 

In  other  words,  the  information  characteristics  were  varied 
separately;  first  the  accuracy  of  the  information  was  varied 
from  0.  5  to  0.  9  with  pertinence  held  constant,  then  the  pertinence 


of  the  information  was  varied  through  the  same  range  with  the 
accuracy  held  constant.  Each  orientation  could  be  considered 
one  "game.  " 

4.  During  each  "game"  at  various  intervals  (which  corresponded  to 
a  preset  number  of  messages  received  by  the  Blue  unit),  the 
computer  was  stopped  and  a  readout  was  called  for  which  would 
give  the  Blue  units'  estimate  of  the  location  of  the  Red  unit.  Each 
readout  gave  a  data  point  in  terms  of  "ability  to  locate  the  enemy.  " 

The  percentage  of  locations  within  8  km  of  and  including  the  area 
actually  occupied  by  Red  forces  was  tabulated,  and  any  improve¬ 
ment  in  ability  to  located  the  Red  unit  accurately  was  graphically 
portrayed  as  a  function  of  increased  information.  This  ability  to 
locate  the  Red  unit  was  used  as  the  criterion  of  effectiveness  for 
the  various  combinations  of  the  information  characteristics. 

Hantzes,  Teunis  and  Harris  also  of  ORO  used  this  method  to  determine 
performance  implication  of  information  requirements  for  combat  intelligence. 
This  group  interrogated  experienced  officers  at  regular  intervals  while  they 
were  actively  engaged  in  a  hand-played  war  game.  This  interrogation  took 
the  form  of  asking  the  officers  the  following  questions  in  relation  to  a  response 
(immediate  action  order)  they  had  made  to  a  specific  message  (intelligence  input) 

1.  What  orders  are  you  giving  as  a  result  of  the  message? 

2.  Why  are  you  giving  them? 

3.  What  units  are  involved? 

4.  When  are  the  affected  units  to  carry  out  the  order? 

5.  Does  this  message  affect  your  plans  ?  If  yes,  how? 

The  intelligence  messages  were  classified  according  to  their  attributes 
(e.  g.  ,  source  which  collected  the  information,  content  and  phase  of  game  when 
it  was  received  by  the  player).  The  immediate  action  orders  of  the  players  were 
also  classified  by  their  attributes  (e.  g.  ,  type  of  action  called  for,  phase  of  game, 
and  reasons  for  giving  the  order).  A  comparison  was  made  relating  responses 
of  the  officers  to  the  attributes  of  the  intelligence  messages,  specifically  to 
(l)  number  of  incoming  messages,  (2)  reasons  for  information  use  and  nonuse, 


-18- 


(3)  the  sources  of  collection,  and  (4)  the  content  of  messages.  From  these 
comparisons  a  list  of  important  considerations  related  to  information  require¬ 
ments  which  could  be  drawn  from  the  analysis  was  derived. 

There  are  many  possibilities  for  using  a  computer  simulation  as  the 
vehicle  for  deriving  a  measur e  -  of- worth  for  various  bits  of  intelligence.  For 
example,  it  would  be  possible  to  make  a  computer  simulation  run  similar  to 
that  of  Moss  and  Harris  whereby  assignments  of  weighting  would  be  given  to 
each  intelligence  input  on  the  basis  of  the  data  points  provided  by  the  graphic 
portrayal  described  above.  It  can  be  seen  that  after  each  bit  of  intelligence 
had  been  given  a  weighting  of  utility  these  weightings  could  be  used  to  discriminate 
quantitatively  among  the  outputs  of  surveillance  systems  which  provide  various 
types  of  intelligence  to  the  commander. 

A  posteriori  investigation  has  the  advantage  of  being  able  to  use  the  ultimate 
criterion  measurement,  e.  g.  ,  the  final  outcome  (succes s  or  defeat)  in  actual 
conflict.  However,  many  times  these  investigations  must  rely  upon  written 
reports  and  when  an  attempt  is  made  to  relate  the  stated  criterion  to  specific 
tactical  decisions  and  the  information  required  for  these  decisions,  the  problem 
of  the  reliability,  of  human  memory  becomes  acute.  This  is  also  complicated 
by  the  diversity  of  evaluations  among  different  officers  concerning  the  importance 
and  value  of  certain  intelligence  information  (and  usually  there  is  no  way  to 
measure  this  diversity). 

This  mode  may  be  used  to  collect  data  from  past  exercises  for  the  purpose 
of  describing  "what  happened"  and  "why,  "  In  such  a  case  an  investigation  of 
exercise  data  (messages)  could  derive  information  by  a  cross  analysis  of  various 
parameters  (e.  g.  ,  sender,  recipient,  time  lapse,  mission,  content,  source,  etc,). 

Several  historical  reconstructions  have  been  made  which  deal  with  certain 
aspects  of  intelligence  system  effectiveness,  RAND  (1956)  reports  such  a  study 
for  the  Fifth  Army  in  Italy  during  World  War  II.  This  report  was  based  on  Allied 
and  captured  German  documents.  Its  findings  on  the  importance  of  various  sources 
of  battlefield  intelligence  and  the  amount  of  information  available  to  the  opposing 
armies  are  tentative,  but  they  do  indicate  the  potential  value  of  historical 
research.  Wohlstettler,  (i960),  presents  a  dramatic  picture  of  the  intelligence 
information  available  preceding  the  attack  on  Pearl  Harbor.  This  book,  how¬ 
ever,  makes  no  effort  to  analyze  the  intelligence  information  required  to  avert 


or  reduce  the  severity  of  the  attack.  An  Army  study  of  identifying  strength 
and  location  also  reconstructs  historically  the  Second  World  War  in  the  light 
of  perceived  enemy  situations  vs.  actual  enemy  situations  as  recorded  by  enemy 
records . 

Formal  experiments  have  the  advantage  of  permitting  the  experimenter 
to  control  and  manipulate  the  specific  variables  (in  this  case  intelligence 
information)  with  which  he  is  concerned.  It  allows  him  to  adhere  rigorously 
to  the  formal  rules  of  the  scientific  method  and  to  perform  sophisticated  analysis 
of  the  results  obtained  by  his  manipulations.  However,  the  experimenter  may 
find  it  difficult  to  enlist  the  cooperation  of  competent  subjects  and  to  find  an 
adequate  criterion  against  which  to  measure  the  performance  of  the  subjects 
once  he  has  them. 
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